Assessing FLOSS Communities: An Experience Report from the QualOSS Project

This paper presents work done in the QualOSS (Quality of Open Source Software) research project, which aims at building a methodology and tools to help in the assessment of the quality of FLOSS (free, libre, open source software) endeavors2. In particular, we introduce the research done to evaluate the FLOSS endeavor communities. Following the Goal-Question-Metric paradigm, QUALOSS describes goals, the associated questions and then metrics that allow to answer the questions. As the number of metrics is high, indicators were defined. An indicator is a color signaling how good a project scores in a given metric or set of metrics. In order to obtain accurate indicators, experts were interviewed, but in future they will be based at least in part on statistical analysis. Also, indicators have been defined following a color rule (green, yellow, red and black), going from the best indicator, green, to the worst indicator, black. After applying the QualOSS methodology to four FLOSS projects, namely, Plone, JavaCC, Swallow, and Maemo, some metrics were identified as not informative enough. However they are useful to provide a first glimpse about a given project. An important lesson learnt is to focus the analysis on just a few repositories, such as source code management systems, which help making the quality model usable. In this line we plan to retrieve more advanced metrics in order to have as many metrics as possible. This will provide several points of view from the same information repository. After this, we will re-create indicators that will help in a make decision context to select the best FLOSS component for a set of given requirements. These set of indicators should be validated using both, statistical analysis and interviews with experts.